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Introduction 



A great amount of human and financial resources are being committed 
to creating basic institutional change in educational systemso However, 
there is rarely a systematic evaluation component built into the change 
program to assess the effectiveness of the change attempt in creating 
long term organizational changeo 

This paper will focus on four specific areas; (1) we will begin 
with definitions of evaluation and evaluation research and with a brief 
review of some approaches to evaluation; (2) the functional consequences 
of evaluation will be discussed; (3) we will then discuss arguments against 
the appropriateness of experimental designs for evaluation, some 
unanticipated consequences of evaluation, and design problems in the 
evaluation of social change; (4) a process evaluation model will be 
outlined. 

Evaluation and Evaluation Research 

The Social science literature offers numerous definitions of evaluation, 
both methodological and conceptual. The methodological definitions, which 
are more common than the conceptual definitions, usually emphasize either 
the stages of the evaluation process or the objectives of evaluation. 
Evaluation objectives, according to Brooks (1965, page 3^), include the 
determination of ; (1) The extent to which the program achieves its goal; 

(2) the relative impact of key program variables; and (3) the role of the 
program as contrasted to external variables. The I960 “Glossary of 
Administrative Terms in Public Health" specifies evaluation as: 

The process of determining the value or amount of success 

in achieving a predetermined objective. It includes at least 



- 2 - 



the following steps; Formulation of the objective, identification of the 
proper criteria to be used in measuring success, determination and 
explanation of the degree of success, recommendations for further program 
stude» 

Riecken's broader definition considers dysfunctional program effects as 
well. Here evaluation is described as “the measurement of desirable and 
undesirable consequences of an action that has been taken in order to forward 
some goal that we value" o (1952, page 4) 

Suchman (1967) notes that the assignment of value to some objective 
and the determination of the degree of success in attaining the valued objective 
are inherent in the evaluation process^ Any intentional social action or 
goal oriented activity can be evaluated, and the evaluation may concern 
dysfunctional, as well as desirable, consequences. Hyman, Wright, and Hopkins 
(1962) emphasize the “social change" aspect of evaluated programs by defining 
evaluation as . the procedures of fact-finding about the results of / 

I 

planned social action", (page 2) Most definitiors of the concept stress the 

I 

relationship between program effectiveness and program goals. Greenberg , 

refers to evaluation as " . . . the procedures by which programs are \ 

studied to ascertain their effectiveness in the fulfillment of goals". 

(1968, page 155) 

Suchman clearly differentiates evatuat ion from evaluation research by 
regarding the former as a goal and the latter as a means for reaching that 
goal. Evaluation is "the general process of Judging the worthwhlleness of 
some activity regardless of the method employed", while evaluation research 
Is "the use of the scientific method for the purpose of making an evaluation". 

(Suchman, 1967, page 30 For the purposes of this paper, Suchman 's formal 
definition of evaluation will be adopted: ", • • the determination . . • of 
the results . • . attained by seme activity . • • designed to accomplish 
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some valued goals or objectives » o" (page 32) This definition encompasses 
four major dimensions of evaluation: The process (the '‘deternilnation") J 
the criteria (the "results"); the stimulus (the "vict ivi ty") , and the 
value (the "obj ect i ve" ) « This definition is flexible enough to allow for 
variation in the methods which can be used in the process. 

Evaluation research , more specifically, refers " , • , to the use of 
the scientific method for collecting data concerning the degree to which 
some specified activity achieves some desired effect", (Suchman, 1969* 
pate - 45 ) Similarly, Hyman and Wright view evaluation research as 

. . those forms of planned social action", in which the fact-finding 
methods yield evidence that is objective, systematic, and comprehensive. 
( 1967 , page 185) 

Evaluation research, as seen within the context of planned change, 

"is a form of research which attempts to provide program administrators 
with accurate information on the consequences of his action". (Caro, 1969* 
page 404) In discussing the evaluation of training programs* Hesseling refe 
to evaluation research as: 

... the procedures to determine the degree to which a training 
programme achieves specified results, both intended and unintended, 
and to determine what elements in the situation or in the methods 
used hamper or foster the process of training . « . Evaluation 
research aims at providing a systematic and comprehensive measure 
of success or failure for training programmes, (1966, page 44) 

All of these above definitions emphasize, to varying degrees, that 
evaluation research is a systematic and comprehensive approach to 
evaluation which utilizes the scientific method. The same procedures 
that were used to discover knowledge are now being used to evaluate one's 
ability to apply that knowledge, (See Suchman, 1967, page 2) The logic 
of basic scientific research Is the same logic used in evaluation research. 
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the difference Is In purpose rather than method o 

Some Approaches to Evaluation Research 

Two different models of evaluation research or program evaluation 
have been conceptualized by Schulberg, Sheldon, and Baker: The goal -attainment 

model and the systems model a(1969« page 6) In the former model, the 
researcher focuses on whether or not the program under study attains a 
prespecified objective* The systems model, which is more open and less limited 
than the goal -attainment model, is based upon the multiple nature of 
organizational goals* In this model, the effectiveness with which any one 
goal is attained must be considered in relation to its influence upon 
attainment of other system goals. 

Most current evaluation research is based upon the goal -attainment 
model, and the following discussion will focus on that model. The goal- / 

attainment approach is highly consistent with the four steps specified by 
the American Public Health Association's operational definition of , 

evaluation, (above, page 2) The process begins with the formulation of V 

objectives, which may be one of the most critical and difficult aspects 
of an evaluation program. In discussing the task of formulating objectives, 

Hyman states: 

The many difficulties ... — the breadth of the thing subsumed 
under a particular objective, the multiple objectives encompassed 
by many programs, the ambiguity inherent in any or all of the 
objectives as stated, and the disagreement as to the objectives — 
are characteristic of many programs and are enough to stagger the 
imagination of the evaluator, (Hyman, Wright, Hopkins, 1962, page 
7) 

Suchman (1967) points out a number of general considerations 
involved in the formulation of objectives. Those important factors include: 

What Is the nature of the content of the object ive?; Who is the target 
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of the program?: When is the desired change to take place?: Are the 

objectives unitary or multiple? What is the desired magni tude of effect?: 
and How is the objective to be attained? (see pages 39-^1) These 
considerations not only relate to the specification of evaluation objectives, 
but also influence the overall design and methods of the research* 

Program objectives are often too global and general to be used as 
operational goals in evaluation research* Commonly, intermediate goals and 
"practical** objectives are used as measurable indices in social research* 
HacMahon, Pugh, and Hutchinson (1961) differentiate between two types of 
evaluation -- evaluation for accompl i shment and evaluation for technique * 

The former is concerned with ultimate type goals; the latter focuses on 
intermediate type goals, in specific, the quality of the events during the 
research* In many cases, the ultimate objectives of social research are so 
complex and difficult to ope rationalize, that technique evaluation is greatly 
more feasible than the difficult accomplishment evaluation. However, the 
evaluator is working with a scale of objectives if he is basing his work on 
an intermediate goal. The evaluation must move up the scale of objectives 
which can be done in only two ways: (1) by proving the intervening 

assumptions through research effort or (2) by assuming their validity 
without full research proof, (James, 1962, page 33) 

After the objectives are selected, the proper criteria to be used In 
measuring success must be identified. Paul (1956) classified criteria "types" 
into three sets; Assessment of effort (energy and effort output); assessment 
of effect (the results of the effort); assessment of process (analysis of 
why and how an effect was achieved). These classifications, conceptualized 
by Paul, have been expanded and modified by a number of theorists. James 
(1961, 1962) dimensional izes criteria under four categories: 
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(1) Evaluation of effect -- relating to the quality and quantity 
of effort o 

(2) Evaluation of performance -- measures the results of the efforts 
in terms of the stated objectives, 

(3) Adequacy of performance — measures the degree to which effective 
performance is adequate to the total amount of need 

(4) Evaluation of efficiency— measul-es output/ input and includes 
the evaluation of alternative methods in terms of cost, 

Suchman (1967) adds a fifth category, the evaluation of process (as 
suggested by Paul's technique objective), which involves the question of 
how and why a program did or did not work® "The analysis of process may be 
made according to four main dimensions dealing with: (1) The attributes 

of the program itself; (2) the population exposed to the program; (3) the 
situational context within which the program takes place; and (4) the 
different kinds of effects produced by the program", (page 67) 

Recent proposals for evaluation in educational systems have similarly 
dimensional! zed evaluation into "types" or "stages". Stuff lebeam (1967), 

In proposing that evaluation be utilized to facilitate educational 
decisions, has defined four types of evaluation: context, input, process, 

and product. Context evaluation involves the continual monitoring of educational 
systems to determine unmet needs and the underlying causes of problems. 

I nput evaluation concerns the assessment of possible solutions and alternatives 
for alleviating system need^ as they are proposed by outside agencies. Once 
an appropriate response is selected, the treatment is subjected to process 
evaluation to determine whether the program is working as expected and to 
identify needed modifications in the program. Product evaluation is concerned 
with the measurement of the program's overall quality and effectiveness in 
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dealing with systems needs* (See Stufflebeam, pages 129-131) 



Stuff lebeam' s approach to evaluation assumes the school district 
perspective rather than the act ion- research program perspective* Similarly, 
Cuba's (1968) emergent evaluation concerns the. collecting and interpreting 
of data relevant to a series of decisions made by school administrators* 

These decisions involve; (1) identifying needs, (2) Identifying a process 
for coping with the need, (3) implementing the program, and (4) "determining 
the treatment's feasibility, quality, effectiveness, and efficiency"* 

(Cuba, 1968, page 57) 

Potential Functional Consequences of Evaluation Research 

In this section, some of the specific anticipated functional consequences 
of evaluation research will be noted. Reasons for evaluating or potential 
functional consequences of the process have been documented by numerous 
theorists, in particular, Suchman (196?), Campbell (1969), Knutson (1961), 
and Evans (I969). Some general functions of evaluation have been implied 
above; so in this section, specific outcomes will be discussed in relation 
to the change team concept* 

1* A primary and expected product of evaluation is the determination 
of the extent to which the program is attaining prespecified objectives. 
(Evans, 1969) This function of evaluation is exemplified by the Cicarelli 
Report (1969) on the impact of Head Start. Evaluation of the change team 
concept can potentially determine the extent to which individual change agents 
have been trained to work effectively in teams, the degree to which teams have 
successfully increased organizational health in target schools, and the 
frequency of innovation adoption in those schools. 

2. Evaluation can potentially improve the change team concept by 
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identifying weaknesses In the training and develo[^ent progranio The 
Cicarelli Report, mentioned above, concluded with suggestions for a 
number of program modifications on the basis of weaknesses identified by 
means of the evaluation^ Such feedback could similarly be utilized to 
modify the change team program to alleviate identified weaknesses » 

3. Evaluation facilitates the comparison of the selected program 
to other methods designed to accomplish similar object iveso Certain 
evaluation efforts concentrate on a particular program and then compare 
the strategy to other approaches designed to alleviate the need, 

(See Gove and Costner, 1969) Change teams could be compared to other 
programs for school development, such as sensitivity training or problem 
solving techniques, and particularly to programs which train individual 
change agents. 

4. Freeman and Sherwood (1965) suggest that evaluation forces 
those responsible for program design to clearly specify their objectives, to 
define what they are trying to do, and to determine what specific changes 
they are trying to effect. This consequence is particularly functional In 
the case of recently conceptualized strategies, such as the change team 
approach. The specification of objectives Is also useful in uncovering 
inconsistencies in objectives and procedures, particularly when the systems 
approach to evaluation is adopted, (See Etzioni, I960; Suchman, 1967) 

5* Evaluation Is potentially useful for educational programs as a 
means for identifying “boomerang'* or unanticipated dysfunctional program 
consequences. Evaluation has been used effectively to uncover unintended 
outcomes related to school system programs (See Messick, 1970) and training 
and development programs (See Hesseling, 1966), It is suggested that 
evaluation of change teams could potentially point out certain dysfunctions 
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of the progranio 

6» Evaluation will indicate the degree of transferability of the change 
teain program to other school districts in other geographical regions and 
socio-economic conditions, (Suchman, 1967) As change team effectiveness under 
particular conditions is reviewed, suggestions will emerge for necessary 
modifications to fit the program to different times and different places* 

7, As the change team concept is subject to evaluation, the scientific 
basis for educational administrative practice will be advanced. Effective 
evaluation will stimulate new hypotheses and generate improved strategies 
concerning educational organization development* Additionally, administrative 
science will be advanced as the effectiveness of new change supporting 
structures are evaluated* (See Suchman, page l4l) 

8. From the societal perspective, a most Important function of 

/ 

evaluation is the provision of public accountability* Evaluation studies 
are helpful in demonstrating to the public that particular programs are 
worth supporting and possible expanding* On the other hand, evaluation also 
provides a mechanism for Identifying unsuccessful or uneconomical programs 
which should be discontinued. It is suggested that public knowledge regarding 
success or failure of education action programs is particularly important* 

Two sectors of the public must be aware of the value of the program; 

(1) The general public, whose taxes support the program and (2) the local 
public, whose schools are participating in the change team strategy* 

9* Effective evaluation will potentially enhance the involvement 
and motivation of all personnel participating in the program. As standards 
and objectives are established, participants are provided an opportunity to 
measure their progress and achievement. Additional is change teams begin 
functioning in target schools, evaluation activities wHl increase communication 
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between the university and the practicing administrators. 

There are, of cciirse, numerous other functional consequences of 
evaluation. For example, the extent to which the program is being conducted 
as originally conceived is measured through continual monitoring. It can 
possibly be determined if the program is precisely aimed at the needs for 
which it was designed to alleviate. Sophisticated evaluation can also provide 
a costs/benefits analysis and assist in determining priorities for human 
effort and available funds. Similarly, slightly different program methods 
and approaches can be compared to assess feasibility and relative 
effectiveness (Knutson, I 961 ) As such, the potential functional consequences 
of evaluation and evaluation research are impressive. However, it has been 
emphasized that these functions are merely potential consequences of program 
evaluation. 

Problems and Potential Dysfunctional Consequences 

Some of the strongest arguments against the suitability of evaluation 
research based on the experimental design for large-scale programs have 
been made by Weiss and Rein In “The Evaluation of Broad-Aim Programs: 
Experimental Design, Its Difficulties and an Alternative". (1971) They suggest 
that in the evaluation of many social action programs, the use of the 
experimental design often creates technical and administrative problems 
so severe as to make the entire evaluation of questionable value. The 
dysfunctional consequences and problems of evaluation research as presented 
by Weiss and Rein and other researchers will be the topic of this section. 

Firstly, there is a general tendency to avoid evaluation. This is 
often due to the anxiety the practitioner perceives, because of the 
ambiguities surrounding the evaluation process. The change- 
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target system may be Investing a lot of its resources in a change program, 
and the evaluation of this effort may be seen as nothing more than a "critical 
spying" to determine whal v/ent wrong with the program, (Weiss, 1972) 

Secondly, it Is diff It to select satisfactory criteria, Broad-alm 
programs often have higi! non-specific goals which do not lend themselves 
to operationalization. The program alms may be specified In many ways and 
It is not always known how the desired changes will manifest themselves. In 
cases such as this, the research team will often attempt to gather pretest 
and post- test data along a large number of variables. The data is searched 
for significant dieferences, and the program is pronounced effective along 
those variables. However, it is not always clear whether these changes 
result from the program or by chance. "Moreover, the variables which show 
diff.^‘"ences are likely to be identified as the successfully achieved 
goals of the program, although, had the system been only a bit different, 
it might well have taken a different path, and other variables might have 
signaled the change", (Weiss and Rein, page 104) (Note: Campbell takes 
issue with the claim that these errors, and others, are intrinsic to the 
experimental method. See Campbell, 1971 •) 

The specification of program objectives and operational criteria for 
measurement restricts the evaluation to anticipated consequences of education 
act ion- research programs. This limitation of evaluation research is very 
serious, especially in the case of large-scale broad-aim social programs. 

In some cases, it is possible that a program will result in certain 
serious dysfunctional consequences that were not anticipated. Though the 
evaluation data may show favorable changes in certain respects, these 
advantages may be outweighed by the concomitant dysfunctions. 

Thirdly, the inherent inflexibility of the experimental design may 
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prevent the evaluation team from being able to adjust to changes in program 
management and policy. Program administrators may slightly change their 
objectives and redirect resources as they become more familiar with their 
situation. As their knowledge of their staff, the target population, and the 
social problem increases, and as they begin to get preliminary feedback on 
action program strategies, administrators can be expected to modify their 
activities. If the modification involves objectives, the evaluator is left 
with an outdated criteria. The presence of an evaluation team may actually 
increase the probability that a program will be changed. Brooks notes that 
the ethical necessity for continuous feedback of research findings acts 
to constrain evaluation in community action programs. (1965, page 61) 

In general, a large-scale program might be flexible enough to operate 
within a dynamic and complex environment; however, it is extremely 
difficult for the experimental design to be as flexible as the program. 

Fourthly, the number of replications of large-scale programs is often 
too small to provide statistically significant samples. Even in relatively 
••small-scale" programs, the number of true replications may be very limited. 
Scriven reports that there is a general lack of a common basic framework in 
educational programs which diminishes the generation of comparable data, 
( 1969 , page kS) 

Fifthly, the applicability of the experimental design Is limited due to 
the fact that treatments are not standardized. (Guba, page <Sl, Weiss and 
Rein, page 105) In the case of the change team training program, program 
procedures would initially be relatively standardized. However, as the 
change teams begin practicing, their activities will be structured on the 
basis of their particular schools' environments. Nevertheless, this problem 
is possibly less serious for change team evaluation than it is for the 
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evaluation of broad-scale community action programs. 

Sixthly, an experimental design is limited in the information it can 
produce. Weiss and Rein assert that numerous programs are destined to failure 
due to general resistance; and an evaluation report of "no success" does not 
contribute much. It Is suggested that evaluation should concentrate on Identi- 
fying the nature of the opposition, specifying reasons for failure (or success), 
and determining the program's unanticipated consequences* 

Seventhly, Cuba suggests that the statistical assumptions of normal 
distributions and additivity are unrealistic for educational research* 

(pages 60-61) The assumption that groups are randomly assigned to 
treatment and control conditions is usually not attainable in education 
act ion- research programs* As such, the internal validity of the evaluation 
is limited and the general izabi 1 ity is also threatened* (See Suchman, 1967> 
and Cuba, 1968) Additionally, the experimental design type of evaluation 
fails to provide for decision making* (Sea Stuff lebeam, 196^7) 

Eighthly, many difficulties in evaluation research and program 
administration result from conflicts between^ the research team and program 
administrators* It seems that there is often a basic distrust between these 
two groups* Beneficial communication is minimal as a result of mutual 
stereotypes in the relationship* (Schulberg and Baker, 1968, page 563) 

Rodman and Kilodny OSSk) found that the practitioners perceive the 
researcher's objectivity as hostile and dislike the questioning attitudes 
of evaluators. This tension is increased as a result of status problems, 
different time orientations, and disagreements over publications* 

The problem of the operationalization of goals often creates further 
tensions between program administrators and evaluators* Often, an evaluator 
may have to monitor a program on the basis of measurable objectives which 
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fail to reflect the true intention of the programo Additionally, the goals of 
educational programs may be long range while evaluation must often focus on 
more immediate changes* An unanticipated consequence of evaluation is that 
these immediate empirically obserbable objectives may become the program goals * 
Program administrators may feel “forced” to direct their attention and 
resources towards measurable program outputs. 

Outline of a Process Evaluation Model 

In the above few pages, we have briefly focused on some of the 
positive and negative aspects of evaluating change programs. For more 
detailed discussion, see Caro (1971) Readings in Evaluation Research and Weiss 
(1972) Evaluating Action Programs; Readings in Social Action and Education . 

Even when considering the problems and potential dysfunctional 
consequences of evaluation, it is apparent that evaluation can be useful to 
the change agent. Process evaluation is useful in that it can provide him 
with a continuing monitory of the change process , of what*s working and not 
working, so that he can modify his interventions when they are having no 
impact on the system . 

This type of process evaluation is geared to provide the change-agent 
or change agent team with feedback to maximize their impact on the change 
target system. In this sense, it is not an experimental evaluation design 
to “test" a specific type of intervention, because the intervention is being 
modified based on the feedback the change agent receives. 

For example, a school system might decide with the help of an outside 
consultant to Institute sensitivity training as part of a leadership 
development program. However, after the sensitivity training begins, it may 
become apparent that there is too much hostility in the system to run 
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effective T-groups« Conflicting groups may first need to be subjected to 
some of the Blake and Mouton (1969) variety* This kind of feedback to the 
change agent is helpful in that it may allow him to implement an 
alternative intervention that might be more effective. 

A specific process evaluation model will now be presented that the 
educational administrator might consider in implementing change programs. 

Both Duncan (1972) and Zaltman and Coughlan (1972) have made a strong case 
for the change agent teams versus the single change-agent. The change agent 
team comprised of internal and external change agents also has the advantage 
in evaluation. By having several change agents, some can concentrate on the 
intervention activity while others can do the evaluation. Dividing the 
intervention and evaluation roles between several change agents has the 
potential for providing a more objective assessment. The interventionist 
is likely to be more biased in assessing what is happening in the system. 
Other members of the team can focus on collecting feedback data that then 
can be transmitted to the interventionist, so that he can modify his behavior 
if need be. 

Specifically, this process evaluation model has several stages. First, 
a change agent team is constructed of a combination of internal and external 
change agents. This team approach has the advantage of combining an outside 
expert with his collection of skills with members of the target system and 
their potentially more thorough knowledge of that system (See Duncan, 1972). 

The second stage is to select one or two representatives from each of 
the work groups in the target system that are going to be affected by the 
change program. These representatives will form an Organizational Change 
Committee. This Organizational Change Committee wil 1 provide a direct 
liaison between the change agents and the rest of the organization. The OC 
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Committee will work with the change agent team in the diagnosis and the 
feedback of the diagnosis to the change target system as Duncan and Radnor 
(1972) have outlined in their institutionalization of change model o 

Once the Organizational Change Committee has been selected, the change 
agent team works with them to specify the process evaluation. The members 
of the OC Committee, by nature of their membership in the system, can 
facilitate the evaluation process in several ways. First, as ongoing 
members of the system, they are in a better position to monitor the 
effects of any change program over a period of time. Secondly, by virtue 
of their membership, they are better able to assess the impact of the change 
program from the "user's’’ standpoint. They are able to help define the 
criteria for evaluation. For example, if a leadership training program was 
instituted, behavioral observation could be utilized In work groups to 
determine the actual changes that took place. The evaluation would not be 
relying on self reported behavioral changes. 

Thirdly, the members of the target system can be helpful in specifying 
the kinds of data collection procedures that may be used. For example, by 
having target system people involved, a broader range of data can be 
collected. Here participant observation could be used to assess actual 
behavioral changes, and, by observing these changes first hand, a better 
understanding of the process by which these changes occurred can be 
identified. The traditional before-du ring-after questionnaire can still be 
utilized, but probably in a more effective way. Because of their involvement 
in the evaluation process, change target members are likely to be less 
resistant and more consciencious in filling out of the instruments. Thus, 
the evaluation is likely to generate more valid data. 

Fourthly, by involving members of the client system through the OC 
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Committee in the evaluation, some of the potential problems in evaluation 
could be avoided* Some of the initial resistance and anxiety surrounding 
evaluation (Weiss, 1972) might be reduced as members of the system are 
involved in specifying the design, and thus feel they have more control 
and a better understanding of the evaluation* In this case, evaluation is 
less likely to be seen as "critical spying"* Also, because of the involvement 
of members in the target system in the evaluation, there would be a greater 
likelihood of the system's utilizing the results of the evaluation in 
modifying change programs* There is also less likelihood for the system's 
members to deny and displace the evaluator and his findings (See Rodman 
and Kolodny, 1964, page 127)* 

Conclusion 

Thus, in this process evaluation model, organizational members, through 
the Organizational Change Committee, work in a collaborative relationship 
with the change agent team in the design and implementation of the 
evaluation* This evaluation technique has two objectives: (1) it provides 

the change agent team with a continuous monitoring of the impact of their 
interventions and has the potential for helping the change agent team modify 
their interventions when they are not effective; (2) it creates a step-by- 
step documentation of the change program and its impact on the system that 
can be used not only to assess the program, but also to plan for future 
changes. It documents what worked and did not work, so that future programs 
can benefit from past experiences. 
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